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EXECUTIVE  SUMMARY 


This  study  has  demonstrated  that  it  is  possible  to  use  the  outputs  from  a  physics-based 
modeler  (magnetometry  data  only)  for  training  a  probabilistic  neural  network  (PNN)  to 
discriminate  UXO  from  scrap.  As  expected,  classification  performance  was  found  to  be 
data  set  dependent.  The  PNN  classification  performance  tracked  the  visual  clustering  of 
the  data  seen  using  principal  component  analysis  and  nonlinear  mapping  (i.e.,  good 
clustering  =  good  classification  performance).  Model  outputs  from  BBR2  were  used  to 
train  a  PNN  model,  which  could  correctly  discriminate  UXO  from  scrap  at  the  BBR1 
location.  The  exact  cause  for  many  of  the  misclassified  UXO  could  not  be  determined 
from  the  results  in  this  study,  although  most  of  the  misclassifications  occurred  for  small 
ordnance.  The  model  outputs  for  the  misclassified  objects  were  much  different  than  the 
outputs  for  the  others  in  that  UXO  class.  Further  improvements  in  the  modeler  or  the  use 
of  data  fusion  of  MAG  and  EM  responses  must  be  made  in  order  for  classification  results 
to  improve. 


iv 


INTRODUCTION 


Locating,  identifying  and  disposing  of  buried  UXO  on  the  10  million  acres  of 
contaminated  lands  in  the  continental  United  States  is  a  500  billion  dollar  problem. 
Development  of  new  technologies  with  improved  data  analysis  has  been  identified  as  a 
high  priority  triservice  requirement.  Using  current  methods,  it  has  been  shown  that  false 
alarm  detections  far  outnumber  correctly  identified  ordnance.  The  best  performing 
technologies  typically  have  a  false  alarm  rate  of  300-500%. l’ 2’  3’  4  The  high  cost  of 
digging  and  disposing  of  targets  accounts  for  the  overwhelming  portion  of  the  costs  of 
UXO  remediation,  therefore  a  substantial  saving  could  be  recognized  if  the  number  of 
false  positives  were  reduced.  Using  data  collected  by  the  Naval  Research  Laboratory’s 
Multi-sensor  Towed  Array  Detection  System  ( MTADS ),  new  software  techniques  are 
being  developed  to  improve  discrimination  and  reduce  the  false  alarm  rate.  The  program 
has  three  parts:  Phase  1,  Target  Detection,  Phase  2,  Quantitative  Modeling,  and  Phase  3, 
Target  Classification.  This  paper  describes  the  first  stages  of  Phase  3,  Target 
Classification. 

A  feasibility  study  on  using  a  physics-based  modeler  to  generate  inputs  for  a 
probabilistic  neural  network  (PNN)  for  UXO  classification  has  been  completed. 
Magnetometer  survey  data  taken  at  three  field  sites  using  the  MTADS  were  used.  The 
physics-based  modeler  (point-dipole  model)  used  in  the  MTADS-DAS  (MTADS-Data. 
Analysis  System)  estimates  the  object’s  depth  (dep),  size  (siz),  inclination  (inc),  magnetic 
moment  (mom),  azimuth  (azi),  and  also  returns  a  modeler  fit  quality  (fit).  The  modeler 
parameters  dep,  mom,  inc,  and  azi  are  independent  variables  while  siz  is  a  dependent 
variable.  The  parameter  fit  is  neither  and  reflects  how  well  the  modeler  was  able  to 
describe  the  target  item  in  question.  A  new  modeler  based  on  oblate  spheroids  is  being 
developed  in  Phase  2  of  this  program  to  improve  shape  discrimination.  The  new  modeler 
was  not  available  for  this  feasibility  study.  In  the  current  MTADS-DAS,  a  trained  analyst 
performs  classification  using  both  visual  clues  and  the  modeler  outputs  as  a  guide.  The 
analyst  also  uses  knowledge  of  site  use  history  and  ground  truth.  This  process  can  be 
slow  and  tedious  even  for  an  expert  analyst  working  with  large  data  sets.  In  order  to 
expedite  this  process,  we  are  studying  the  feasibility  of  using  a  PNN  to  automate  the 
UXO  discrimination  task. 

In  this  study,  the  discrimination  of  ordnance  from  ordnance  scrap  items  and  other 
clutter  for  three  different  locations  was  performed  using  the  PNN.  After  evaluating  the 
initial  classification  results,  the  data  sets  were  characterized  and  several  strategies  were 
investigated  to  improve  the  performance  of  the  PNN  on  these  data  sets. 
Recommendations  are  then  given  for  further  research  in  this  area. 

EXPERIMENTAL 

Outputs  from  the  physics-based  modeler  and  associated  dig  sheets  were  received 
in  Microsoft  Excel  spreadsheets  and  read  into  Matlab  (Version  5.2,  Mathworks,  Inc., 
Natick,  MA)  as  a  tab-delimited  text  file.  These  model  outputs  are  given  in  Tables  7, 5  10, 
and  ll.6  Prior  to  PNN  training,  the  raw  model  outputs  were  autoscaled  (columns  with  a 
mean  of  zero  and  unit  variance).  The  PNN  software  and  nonlinear  mapping  routines 
were  written  by  the  authors  in  Matlab.  Principal  Component  Analysis  (PCA)  was 


implemented  using  the  PLS_toolbox,  version  2.0c  (Eigenvector  Technologies,  Inc., 

Manson,  WA).  .  , 

The  factors  that  describe  the  major  trends  in  the  data  can  be  evaluated  through 

PCA.  One  of  the  most  useful  first  steps  in  multivariate  analysis  is  to  observe  the 
clustering  of  the  data  in  the  multi-dimensional  space.  Because  it  is  impossible  to 
visualize  the  data  points  clustering  in  n-dimensional  space,  display,  mapping  and  cluster 
analyses  are  used.  An  exploratory  algorithm  was  used  in  this  study  to  provide  an 
interpretable  view  of  the  multi-dimensional  data  space.  Principal  Component  Analysis 
(PCA),7  also  known  as  the  Karhunen-Loeve  transformation,  is  a  display  method  that 
transforms  the  data  into  two-  and  three-dimensional  space  for  easier  visualization.  PCA 
finds  the  axes  in  the  data  space  that  account  for  the  major  portion  of  the  variance  while 
maintaining  the  least  amount  of  error.  PCA  finds  the  linear  combinations  of  variables  or 
sensors  that  describe  the  major  trends  in  the  data.  The  first  principal  component  captures 
the  largest  amount  of  information  or  variance  in  the  data.  The  best  plane  that  represents 

the  data  space  is  achieved  by  plotting  the  first  two  principal  components. 

Mathematically,  PCA  computes  a  variance-covariance  matrix  for  the  stored  data 
set  and  extracts  the  eigenvalues  and  eigenvectors.  PCA  decomposes  the  data  matrix  as 
the  sum  of  the  outer  product  vector,  referred  to  as  loadings  and  scores.  The  scores 
contain  information  on  how  the  targets  are  related  to  each  other  and  the  loadings  contain 
information  on  how  the  variables  or  model  outputs  relate  to  each  other.  Examination  of 
the  results  of  these  methods  provides  insight  into  the  data  set.  Non-linear  mapping  is 
used  to  further  improve  the  2-dimensional  display.  This  routine  seeks  to  retain  the 
distances  between  the  data  points  in  n-dimensional  space  in  the  2-dimensional  space  by 
minimizing  the  mapping  error. 

Discrimination  of  UXO  and  clutter  are  being  investigated  in  this  program  using 
multivariate  classification  methods.  Classification  methods  are  supervised  learning 
techniques  that  use  training  sets  to  develop  classification  rules.  The  rules  are  used  to 
predict  classification  of  a  future  data  set  or  site.  These  methods  are  given  both  the  data  and 
the  correct  classification  results  (i.e.,  “ground  truth”),  and  they  generate  mathematical 
functions  to  define  the  classes.  The  best  classification  algorithms  are  those  that  provide  the 

best  prediction.  .  ...  , 

The  PNN  method9, 10  was  used  in  this  study  because  it  provides  a  probability  that 

the  target  class  is  present  and  the  level  of  confidence  can  be  adjusted  according  to  the 
ultimate  land  use  requirements.  Probabilistic  Neural  Networks  are  a  class  of  neural 
networks  that  combine  some  of  the  best  attributes  of  statistical  pattern  recognition  methods 
and  feed-forward  neural  networks.11,  12  They  have  been  described  as  the  neural  network 
implementation  of  kernel  discriminant  analysis  and  were  first  introduced  into  the  neural 
network  literature  by  Donald  Specht  in  the  late  1980's.11  Initially  developed  for  radar 
classification,  the  PNN  has  been  used  in  a  wide  variety  of  applications  including  fingerprint 
identification12,  optical  character  recognition,  remote  sensing,  image  processing,  gas 
chromatography,16  and  chemical  sensor  arrays.17 

Figure  1  shows  the  architecture  of  the  PNN.  The  PNN  is  a  nonlinear,  nonparametnc 
pattern  recognition  algorithm  that  operates  by  defining  a  probability  density  function  (PDF) 
for  each  data  class  based  on  the  training  set  data  and  the  optimized  kernel  width  parameter. 
For  ordnance  discrimination,  the  inputs  are  the  physics-based  modeler  outputs  or  pattern 
vectors.  The  outputs  are  the  Bayesian  posterior  probability  (i.e.,  a  measure  of  confidence  in 
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the  classification)  that  the  input  pattern  vector  is  a  member  of  one  of  the  possible  output 
classes,  for  example,  UXO  or  scrap. 
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Figure  1.  Topology  of  a  PNN.  Modeler  outputs  are  used  as  input  and  the 
probability  of  belonging  to  one  of  the  specified  classes  is  determined 


The  hidden  layer  of  the  PNN  is  the  heart  of  the  algorithm.  During  the  training 
phase,  the  pattern  vectors  in  the  training  set  are  simply  copied  to  the  hidden  layer  of  the 
PNN.  Unlike  other  types  of  artificial  neural  networks,  the  basic  PNN  only  has  a  single 
adjustable  parameter.  This  parameter,  termed  sigma  (a),  or  kernel  width,  along  with  the 
members  of  the  training  set,  define  the  PDF  for  each  data  class.  In  a  PNN,  each  PDF  is 
composed  of  Gaussian-shaped  kernels  of  width  ct  located  at  each  pattern  vector.  The  PDF 
essentially  determines  the  boundaries  for  classification.  The  kernel  width  is  critical  because 
it  determines  the  amount  of  interpolation  that  occurs  between  adjacent  pattern  vectors.  As 
the  kernel  width  approaches  zero,  the  PNN  essentially  reduces  to  a  nearest  neighbor 
classifier.  This  point  is  illustrated  by  the  contour  plot  in  Figure  2.  These  plots  show  four, 
two-dimensional  pattern  vectors  for  two  classes  (A  and  B).  The  PDF  for  each  class  is 
shown  as  the  circles  of  decreasing  intensity.  The  probability  that  a  pattern  vector  will  be 
classified  as  a  member  of  a  given  output  data  class  increases  the  closer  it  is  to  the  center  of 
the  PDF  for  that  class.  In  this  example,  any  pattern  vectors  that  occur  inside  the  inner-most 
circle  for  each  class  would  be  classified  with  nearly  100%  certainty.  As  a  is  decreased 
(upper  plot),  the  PDF  for  each  class  shrinks.  For  very  small  kernel  widths,  the  PDF  consists 
of  groups  of  small  circles  scattered  throughout  the  data  space.  A  large  kernel  width  (lower 
plot)  has  the  advantage  of  producing  a  smooth  PDF  and  good  interpolation  properties  for 
predicting  new  pattern  vectors.  Small  kernel  widths  reduce  the  amount  of  overlap  between 
adjacent  data  classes.  The  optimized  kernel  width  must  strike  a  balance  between  a  a  that  is 
too  large  or  too  small. 
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Sensor  1 


Figure  2.  Contour  plot  illustrating  the  PDF  for  each  class.  Two  classes 
are  shown,  A  and  B.  Four  events  of  each  type  are  presented.  The  PDF 
for  each  class  is  shown  as  circles  of  decreasing  intensity 
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Prediction  of  new  targets  using  a  PNN  is  more  complicated  than  the  training  step. 
Each  member  of  the  training  set  of  pattern  vectors  (i.e.,  the  patterns  stored  in  the  hidden 
layer  of  the  PNN  and  their  respective  classifications),  and  the  optimized  kernel  width  are 
used  during  each  prediction.  As  new  pattern  vectors  are  presented  to  the  PNN  for 
classification,  they  are  serially  propagated  through  the  hidden  layer  by  computing  the 
Euclidean  distance,  d,  between  the  new  pattern  and  each  pattern  stored  in  the  hidden  layer. 
The  Euclidean  distance  scores  are  then  processed  through  a  nonlinear  transfer  function  (the 
Gaussian  kernel): 

Hidden_Neuron_Output  =  expi-idyo2)  (1) 

Because  each  pattern  in  the  hidden  layer  is  used  during  each  prediction,  the  execution  speed 
of  the  PNN  is  considerably  slower  than  some  other  algorithms.  The  mass  data  storage 
requirements  can  also  be  quite  large  since  every  pattern  in  the  hidden  layer  is  needed  for 
prediction. 

The  summation  layer  consists  of  one  neuron  for  each  output  class  and  simply 
collects  the  outputs  from  all  hidden  neurons  of  each  respective  class.  The  products  of  the 
summation  layer  are  forwarded  to  the  output  layer  where  the  estimated  probability  of  the 
new  pattern  being  a  member  of  each  class  is  computed.  In  the  PNN,  the  sum  of  the  output 
probabilities  equals  100%.  Using  the  model  outputs  as  the  input  training  set,  PNNs  were 
trained  to  perform  the  classification.  One  of  the  main  advantages  associated  with  using  a 
PNN  is  the  ability  to  output  a  probability  for  each  of  its  classifications.  For  critical 
applications,  such  as  ordnance  detection  and  remediation,  such  an  indicator  of  confidence 
is  extremely  useful  in  assisting  the  decision  making  process  and  reducing  the  likelihood 
that  individual  ordnance  items  are  missed  by  reducing  the  detection  probability. 

As  discussed  above,  the  calculation  of  the  optimum  kernel  width,  a,  is  imperative 
for  high  classification  rates  to  be  achieved.  For  the  work  described  herein,  an  iterative 
algorithm  for  a  optimization  was  employed.9  The  algorithm  was  designed  to  minimize 
the  following  cross  validation  error: 


error 


=  A',„+X 


a -p,)2+5>,)2 

i*j 


(2) 


where  pi  is  the  predicted  probability  of  being  the  correct  class,  pj  is  the  predicted 
probability  of  all  the  other  classes,  Nm  is  the  number  misclassified  and  n  is  the  number  of 
patterns  in  the  training  set.  The  minimization  of  Equation  2  is  based  upon  a  parabolic 
interpolation  method  described  by  Masters.18 
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RESULTS  AND  DISCUSSION 


Badlands  Bombing  Range  (BBR  1) 

A  subset  of  the  Badlands  data  set,  Bull’s  eye  Target  1  (Pine  Ridge  Reservation, 
BBR  1)  was  investigated  to  determine  the  ability  of  a  PNN  to  distinguish  unexploded 
ordnance  (UXO)  from  scrap.  For  PNN  classification,  the  Badlands  data  set  consisted  of 
Magnetometer  (MAG)  model  outputs  from  87  buried  objects  that  had  been  remediated.  A 
87  x  6  data  matrix  was  formed  where  each  row  consisted  of  a  set  of  model  outputs  for  a 
target.  The  6  columns  are  the  parameters  derived  from  the  physics-based  modeler.  Each 
row  represents  a  “pattern  vector,”  using  pattern  recognition  terminology.  These  pattern 
vectors  were  subdivided  into  two  categories  or  classes;  UXO  and  scrap.  The  UXO  subset 
consisted  of  44  distorted  and  intact  M3  8  bombs,  while  the  scrap  subset  primarily 
consisted  of  an  assortment  of  bomb  tail  fins,  dry  holes,  and  other  ordnance  scrap  totaling 
43  targets.  The  PNN  was  used  to  discriminate  the  two  classes  of  objects  (i.e., 
differentiate  UXO  from  scrap).  This  data  set  is  limited  in  scope,  representing  a  simple 
test  case,  as  it  contains  only  one  type  of  UXO.  Furthermore,  much  of  the  scrap  is  located 
near  the  surface  while  the  bombs  are  generally  found  further  below  the  surface.  The 
parameters  used  in  this  work  were  derived  from  magnetometry  data,  which  is  different 
from  the  magnetometry  and  time-domain  pulsed  induction  sensor  joint  inversion  modeler 
data  that  will  be  used  in  the  future.  However,  despite  these  limitations,  the  BBR  1  data 
set  provides  an  excellent  platform  for  determining  the  feasibility  of  using  model  outputs 
for  PNN  classification. 

Data  Set  Characterization 

An  examination  of  the  data  set  using  PCA  and  non-linear  mapping  was 
undertaken  to  understand  the  clustering  and  overlap  of  the  UXO  classes  and  scrap  in  the 
data  set.  A  plot  of  the  first  two  factors,  or  principal  components,  for  the  BBR  1  data  set  is 
shown  Figure  3.  The  classes  are  reasonably  separated  from  one  another,  although  there  is 
some  overlap  and  two  UXO  targets,  19  and  34  are  located  within  the  scrap  cluster.  A  plot 
of  the  data  in  two  dimensions  generated  by  non-linear  mapping  is  shown  in  Figure  4.  An 
examination  of  Figure  4  reveals  results  similar  to  those  in  the  PCA  plot.  In  both  plots,  the 
scrap  and  M38  classes  have  some  overlap  (4  and  3  scrap  targets  found  in  the  M38  data 
space  for  PCA  and  non-linear  map  plots  respectively). 
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Cross-Validation  Performance 


PNN  classification  performance  was  examined  using  leave-one-out-cross- 
validation  (CV).  Performance  evaluation  using  cross-validation  involves  training  the 
PNN  using  all  the  patterns  in  the  data  set  except  one,  which  is  withheld  for  subsequent 
prediction.  This  process  is  repeated;  leaving  out  each  pattern  in  turn  until  all  patterns 
have  been  used  once  in  prediction.  In  general,  CV  is  preferred  to  using  a  separate 
external  validation  or  prediction  set  when  only  a  small  data  set  is  available.  In  this  work, 
the  PNN  returned  two  values  for  each  pattern:  the  probability  of  being  UXO  and  the 
probability  of  being  scrap.  Each  pattern  was  assigned  to  the  category  having  the  highest 
probability.  In  a  real-world  scenario,  this  probability  threshold  can  be  adjusted  for  a 
particular  site  using  a  priori  information  regarding  the  types  and  relative  number  of 
buried  objects,  as  well  as  the  level  of  clearing  desired  for  the  ultimate  land  use 
requirements. 

To  determine  which  MAG  model  parameters  were  most  important  and  to  gain 
insight  into  the  PNN  classification,  the  effect  of  various  model  parameters  were  studied. 
PNN  CV  performance  was  determined  for  all  possible  combinations  of  one,  two,  three, 
four,  five,  and  six  MAG  model  parameters.  Results  from  the  single,  five  and  six 
parameter  patterns  and  representative  results  from  the  two,  three,  and  four  parameter 
patterns  are  given  in  Table  1.  The  UXO  miss  rate  (missed  detection)  is  defined  as  the 


Table  1.  MTADS  parameter  combinations  and  PNN  cross  validation  results  for 
Badlands  data 


1 

2 

Parameter 

Combination 

3  4  5 

6 

UXO  Miss 
Rate  (%)a 

False  Alarm 
Rate  (%) 

#UXO  #  Scrap 
Missed  Missed 

dep 

. 

. 

11.4 

9.3 

5 

4 

- 

siz 

- 

- 

- 

- 

6.8 

16.3 

3 

7 

- 

- 

mom 

- 

- 

- 

9.1 

9.3 

4 

4 

- 

- 

- 

inc 

- 

- 

9.1 

16.3 

4 

7 

- 

- 

- 

- 

azi 

-Q 

o 

d 

37.2 

0b 

16 

- 

- 

- 

- 

- 

fit 

18.2 

39.5 

8 

17 

siz 

inc 

4.5 

7.0 

2 

3 

- 

- 

mom 

inc 

- 

- 

6.8 

7.0 

3 

3 

dep 

- 

- 

inc 

- 

- 

9.1 

9.3 

4 

4 

- 

siz 

mom 

- 

- 

- 

6.8 

14.0 

3 

6 

- 

mom 

- 

- 

fit 

13.6 

7.0 

6 

3 

- 

- 

- 

- 

azi 

fit 

25.0 

23.3 

11 

10 

dep 

siz 

inc 

_ 

_ 

4.5 

7.0 

2 

3 

- 

siz 

mom 

inc 

- 

- 

4.5 

7.0 

2 

3 

- 

siz 

- 

inc 

azi 

- 

4.5 

7.0 

2 

3 

- 

- 

- 

inc 

azi 

fit 

6.8 

11.6 

3 

5 

dep 

siz 

mom 

- 

- 

- 

6.8 

14.0 

3 

6 

- 

- 

mom 

- 

azi 

fit 

18.2 

9.3 

8 

4 

siz 

inc 

azi 

fit 

4.5 

7.0 

2 

3 

dep 

siz 

mom 

inc 

- 

- 

6.8 

7.0 

3 

3 

- 

siz 

mom 

inc 

azi 

- 

6.8 

9.3 

3 

4 

dep 

siz 

- 

- 

azi 

fit 

9.1 

11.6 

4 

5 

- 

siz 

mom 

- 

azi 

fit 

9.1 

18.6 

4 

8 

siz 

mom 

inc 

azi 

fit 

6.8 

7.0 

3 

3 

dep 

- 

mom 

inc 

azi 

fit 

9.1 

9.3 

4 

4 

dep 

siz 

- 

inc 

azi 

fit 

6.8 

7.0 

3 

3 

dep 

siz 

mom 

- 

azi 

fit 

9.1 

11.6 

4 

5 

dep 

siz 

mom 

inc 

- 

fit 

6.8 

9.3 

3 

4 

dep 

siz 

mom 

inc 

azi 

- 

6.8 

7.0 

3 

3 

dep 

siz 

mom 

inc 

azi 

fit 

6.8 

7.0 

3 

3 

Using  a  probability  cut  off  of  50  % 

Favorable  result  is  due  to  over  training  not  intrinsic  properties  of  the  data  and  represents  an  anomaly 
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ratio  of  the  number  of  UXO  classified  as  scrap  to  the  total  number  of  UXO  targets 
classified.  The  false  alarm  rate  is  defined  as  the  ratio  of  the  number  of  scrap  classified  as 

UXO  to  the  total  number  of  scrap  classified. 

The  UXO  miss  rate  and  false  alarm  rate  using  all  parameters  were  6.8%  and  7.0% 
respectively,  corresponding  to  3  UXO  and  3  scrap  targets  missed.  The  parameters  found 
to  be  most  important  (causing  the  greatest  increase  in  UXO  miss  rate  and  false  alarm  rate 
when  left  out)  in  the  tests  using  five  parameter  patterns  were  inc  >  siz  >  azi.  The 
parameters,  dep,  mom,  and  fit  had  no  effect  on  the  performance  when  left  out  of  the 
patterns. 

The  best  single  parameter  for  overall  prediction  was  mom  with  a  UXO  miss  rate 
of  9.1%  and  a  false  alarm  rate  of  9.3%.  While  mom  did  not  negatively  impact  the 
classification  performance  of  the  PNN  when  left  out  of  the  input  patterns,  it  was  the  best 
performing  single  parameter.  This  is  due  to  the  interdependence  of  the  variables  and  PNN 
classification.  Either  inc  or  siz  used  alone  produced  generally  poorer  results:  inc  had  a 
UXO  miss  rate  of  9.1  %  and  a  false  alarm  rate  of  16.3  %  and  siz  had  a  UXO  miss  rate  of 
6.8%  and  a  false  alarm  rate  of  16.3  %.  The  best  UXO  miss  rate  was  obtained  using  siz 
alone  at  the  expense  of  the  false  alarm  rate.  It  should  be  noted  that  the  favorable  result 
using  azi  alone  (UXO  miss  rate  of  0%)  is  an  anomaly  related  to  network  over-training 
and  is  not  indicative  of  any  intrinsic  properties  of  this  parameter.  The  false  alarm  rate 
using  azi  alone  was  37.2  %  indicating  poor  predictive  capabilities.  Finally,  the  use  of  fit 
alone  resulted  in  the  largest  classification  errors:  18.2  %  and  39.5  %  for  the  UXO  miss 
rate  and  false  alarm  rate  respectively. 

The  best  classification  results  were  obtained  using  only  two  parameters,  siz  and 
inc.  This  combination  produced  a  UXO  miss  rate  of  4.5  %  and  a  false  alarm  rate  of 
7.0%,  which  corresponds  to  2  UXO  and  3  scrap  targets  missed  or  94  %  correct 
classification  of  the  data  set.  These  two  parameters  were  also  found  to  be  the  two  most 
important  in  the  five  parameter  pattern  tests.  Increasing  the  number  of  model  parameters 
did  not  improve  the  performance.  Figure  5  contains  a  plot  of  the  PNN  CV  predicted 
probability  of  being  UXO  for  each  of  the  87  patterns  in  the  Badlands  data  sets  using  just 
two  MAG  parameters  (siz  and  inc).  The  solid  line  represents  the  conventional  50% 
probability  cut  off  threshold.  As  discussed  above,  the  probability  threshold  can  be 
adjusted  in  case  a  priori  information  is  available  or  to  reduce  the  number  of  missed 
detections.  However,  in  this  case,  the  two  missed  detections  (objects  19  and  34)  were 
predicted  with  very  low  probabilities  of  being  UXO  (<  10%).  The  misclassified  targets 
were  a  distorted  M38  (target  19)  and  a  burster  and  fuse  assembly  (target  34).  The  three 
scrap  items  were  a  bomb  fin  (52),  and  two  dry  holes  (62  and  63).  These 
misclassifications  occur  because  no  other  UXO  patterns  in  the  Badlands  data  set  have 
similar  MAG  model  outputs.  In  the  eventual  application  of  this  technology,  larger  data 
sets  comprised  of  UXO  and  scrap  from  many  sites  will  be  used  to  prevent  this  situation 
from  occurring.  The  average  probability  of  being  UXO  value  of  the  correctly  classified 
targets  (UXO  and  scrap)  and  the  outliers  are  found  in  Table  2. 
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Figure  5.  PNN  probability  of  being  UXO  for  badlands  data  determined  using  only  siz 
and  inc.  The  misclassified  targets  were  19:  distorted  M38,  34:  M38  burster  and  fuse 
assembly,  52:  bomb  fin,  62:  dry  hole,  63:  dry  hole.  The  first  44  targets  are  UXO 


Table  2.  Probability  of  being  UXO  for  correctly 
classified  UXO,  scrap  and  outliers  in  Figure  5 


Probability 
of  being  UXO 


UXO 

0.937 

Scrap 

0.067 

19 

0.051 

34 

0.041 

52 

0.915 

62 

0.860 

63 

0.794 

a  average  of  correctly  classified  targets 
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The  complete  performance  characterization  of  the  PNN  for  this  dataset  can  be 
seen  by  examining  a  receiver  operator  characteristic  (ROC)  curve19  shown  in  Figure 
6.  Each  point  in  the  ROC  plot  represents  a  different  probability  threshold  at  which 
the  UXO  detection  rate  and  false  alarm  rates  were  calculated.  The  level  of  UXO 
classification  required  for  the  particular  site  in  question  can  be  selected  and  the 
corresponding  false  alarm  rate  for  the  PNN  and  site  is  given  by  the  curve. 


Figure  6.  Receiver  operator  characteristic  curve  for  BBR  1  using  [dep  siz  inc] 


An  extensive  examination  of  the  effects  of  using  different  combinations  of 
modeler  output  parameters  on  the  classification  performance  of  a  PNN  has  been 
completed.  The  results  obtained  in  this  study  are  probably  data  set  dependent  and  may 
vary  according  to  the  type  of  UXO  and  scrap  encountered,  the  amount  of  scrap  present, 
soil  type,  etc.  However,  at  this  location,  only  two  or  three  parameters,  [siz  and  inc]  or 
[dep  siz  inc],  were  necessary  to  correctly  classify  all  but  2  UXO  and  3  scrap  targets. 
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Badlands  Bombing  Range  (BBR  2) 


A  second  subset  of  the  Badlands  data  set,  Bull’s  eye  Target  2  (Badland  Bombing 
Range,  BBR  2,  Pine  Ridge  Reservation)  was  investigated  to  determine  the  ability  of  a 
PNN  to  distinguish  UXO  from  scrap  at  a  site  containing  more  than  one  type  of  ordnance. 
For  PNN  classification,  the  BBR  2  data  set  tested  contained  MAG  physics-based  modeler 
outputs  for  243  buried  objects  that  had  been  remediated.  A  243  x  6  data  matrix  was 
generated.  Each  row  in  the  data  matrix  represents  the  model  outputs  for  a  target  and  is 
referred  to  as  a  pattern  vector.  For  this  study  two  approaches  were  investigated.  In  one 
case,  these  pattern  vectors  were  subdivided  into  two  categories  or  classes,  UXO  and 
Scrap  for  the  data  set  classification  labeled  BBR  2  two-class  problem.  In  the  other  case, 
the  pattern  vectors  were  subdivided  into  four  classes  for  the  data  set  classification  labeled 
BBR  2  four-class  problem.  The  UXO  subsets  of  BBR  2  consisted  of  14  2.75-inch 
Warheads  (2.75  WH),  21  M38  bombs,  27  SCAR  rockets  (2.25-inch  rockets),  while  the 
scrap  subset  (181  targets)  contained  a  wide  variety  of  items  ranging  from  ordnance 
related  scrap  to  fence  wire.  The  data  sets  used  for  classification  testing  were  prepared  by 
selecting  those  targets  with  the  descriptions  listed  above  from  the  modeler  output  and 
truth  tables  for  this  site.  The  PNN  was  used  to  discriminate  either  two  classes  or  four 
classes  of  objects  (i.e.,  differentiate  three  UXO  types  from  each  other  and  scrap).  The 
variability  of  the  modeler  parameters  contributes  to  the  difficulty  in  classification.  The 
parameters  used  in  this  work  were  derived  from  MAG  data,  which  is  different  from  the 
MAG/EM  joint  inversion  modeler  data  that  will  be  used  in  the  future.  The  BBR  2  data  set 
represents  a  challenge  to  the  PNN  classification  routines  using  modeler  outputs  for  the 
discrimination  of  UXO  from  scrap. 

Data  Set  Characterization 

An  examination  of  the  data  set  using  PCA  and  non-linear  mapping  was 
undertaken  to  understand  the  clustering  and  overlap  of  the  UXO  classes  and  scrap  in  the 
data  set.  A  plot  of  the  first  two  factors,  or  principal  components,  for  the  BBR  2  data  set 
are  shown  Figure  7.  Each  of  the  UXO  classes  are  reasonably  separated  from  one  another 
but  there  is  significant  overlap  between  the  the  2.75  WH  and  SCAR  UXO  and  the  scrap 
items.  The  M38  bombs  are  separated  from  the  other  two  UXO  classes  but  more 
importantly  are  better  separated  from  the  scrap.  A  plot  of  the  data  in  two  dimensions 
generated  by  non-linear  mapping  is  shown  in  Figure  8.  An  examination  of  Figure  8 
reveals  that  the  scrap  class  is  heavily  overlapped  with  the  2.75  WH  and  SCAR  UXO 
classes  while  the  M3  8  class  is  somewhat  separated,  similar  to  the  PCA  plot.  In  both  the 
PCA  and  non-linear  map  plots,  the  SCAR  and  M3  8  UXO  classes  have  some  overlap  (6 
and  7  SCAR  targets  in  the  M3  8  data  space  for  PCA  and  non-linear  map  plots 
respectively). 
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PCI 

Figure  7.  PCA  scores  plot  of  the  BBR  2  data  set.  Two  principal  components 
account  for  60.6  %  of  the  variance 


Dimension  1 

Figure  8.  Non-linear  map  plot  of  the  BBR  2  data  set 
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Cross-Validation  Performance 


PNN  classification  performance  was  examined  using  leave-one-out-cross- 
validation  (CV).  In  this  work,  the  PNN  returned  either  four  or  two  values  for  each 
pattern:  the  probability  of  being  one  of  the  three  UXO  types  and  the  probability  of  being 
scrap  or  simply  UXO  or  scrap.  Each  pattern  was  assigned  to  the  category  having  the 
highest  probability.  In  a  real-world  scenario,  this  probability  threshold  can  be  adjusted 
for  a  particular  site  using  a  priori  information  regarding  the  types  and  relative  number  of 
buried  objects. 

To  determine  which  MAG  model  parameters  were  most  important  and  to  gain 
insight  into  the  PNN  classification,  the  effect  of  various  model  parameters  were  studied. 
PNN  CV  performance  was  determined  for  all  possible  combinations  of  one,  two,  three, 
four,  five,  and  six  MAG  model  parameters.  The  UXO  miss  rate  (missed  detection)  is 
defined  as  the  ratio  of  the  number  of  UXO  classified  as  scrap  to  the  total  number  of  UXO 
targets  classified.  The  false  alarm  rate  is  defined  as  the  ratio  of  the  number  of  scrap 
classified  as  UXO  to  the  total  number  of  scrap  classified. 


BBR  2  Classification  (2  Class  Problem) 

The  overall  results  from  the  single,  five  and  six  parameter  patterns  and 
representative  results  from  the  two,  three,  and  four  parameter  patterns  are  given  in  Table 
3  for  a  cutoff  of  50%.  The  UXO  miss  rate  and  false  alarm  rate  using  all  parameters  were 
37.1%  and  4.4%  respectively,  corresponding  to  23  UXO  and  8  scrap  targets  missed. 

The  best  single  parameter  for  overall  prediction  was  siz  with  a  UXO  miss  rate  of 
30.6%  and  a  false  alarm  rate  of  8.8%.  The  use  of  fit  alone  resulted  in  the  largest  false 
alarm  rate:  25.8%  and  52.5%  for  the  UXO  miss  rate  and  false  alarm  rate  respectively. 
Using  a  four-parameter  pattern  containing  dep,  siz,  mom,  inc,  the  UXO  miss  rate  was 
reduced  to  24.2%  and  the  false  alarm  rate  was  9.9%  for  an  overall  correct  classification 
of  86%  of  the  data  set.  When  mom  is  removed  from  this  set,  the  overall  performance  is 
the  same,  while  the  number  of  missed  UXO  are  reduced  to  13  from  15.  The  best  UXO 
detection  was  achieved  with  [siz  inc  azi]:  17.7  %,  and  13.3  %  for  the  UXO  miss  rate  and 
false  alarm  rate  respectively. 
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Table  3.  Parameter  combinations  and  PNN  cross  validation  results  for  BBR  2  data 
set  using  2  classes  (UXO  and  Scrap)  using  a  50  %  cutoff 


dep  siz  mom  inc  azi  fit 


37.1 


4.4 


23 
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The  probability  of  being  a  UXO  is  shown  in  Figure  9  for  all  243  targets  in  the 
BBR  2  data  set.  These  probabilities  were  generated  using  the  three-parameter 
combination  found  to  produce  the  near  best  classification  results:  [dep  siz  inc],  consistent 
with  the  other  datasets.  The  majority  of  the  UXO  have  a  very  high  probability  of  being 
ordnance,  while  the  majority  of  the  scrap  items  have  a  very  low  probability  of  being 
ordnance.  The  upper  solid,  horizontal  line,  at  20%,  represents  an  arbitrary  selection  of  the 
probability  cut  off  to  be  used  for  this  class  and  data  set.  As  discussed  above,  the 
probability  threshold  can  be  adjusted  in  case  a  priori  information  is  available  or  to  reduce 
the  number  of  missed  UXO.  Below  the  20%  probability  cut  off,  1  UXO  target  was 
missed  and  69  scrap  items  were  identified  as  UXO  (false  alarm).  To  reduce  the  number  of 
missed  UXO,  the  probability  cut  off  can  be  reduced  to  10  %,  allowing  detection  of  the 
remaining  UXO  target.  The  cost  of  the  additional  UXO  detection  is  an  increase  in  the 
number  of  false  alarms:  100  false  alarms  versus  69  false  alarms  at  the  20%  probability 
cut  off.  These  results  represent  a  large  improvement  over  typical  methods  (300-500% 
false  alarm  rate)  and  would  result  in  at  least  a  50%  reduction  in  the  number  of  false 
alarms.  The  effects  of  reducing  the  probability  threshold  are  summarized  in  Table  4.  The 
ROC  curve  for  BBR  2  classified  using  two  classes  is  shown  in  Figure  10. 


Figure  9.  Probability  of  being  UXO  versus  target  number  for  the  two  class 
problem  in  the  BBR  2  data  set.  The  first  62  targets  are  UXO 
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Table  4.  Summary  of  UXO  Miss  Rates  and  False  Alarms  for  the  two  class  problem 


Class 

Probability 
Cut  off  (%) 

#uxo 

Misclassified 

UXO  Miss 
Rate  % 

#  False  Alarms 

False  Alarms  (%) 

UXO 

10 

0 

0.0 

100 

55.2 

UXO 

20 

1 

1.6 

69 

38.1 

Figure  10.  Receiver  operator  characteristic  curve  for  BBR  2 


BBR  2  Classification  (4  Class  Problem) 

The  overall  results  from  the  single,  five  and  six  parameter  patterns  and 
representative  results  from  the  two,  three,  and  four  parameter  patterns  are  given  in  Table 
5.  The  UXO  miss  rate  and  false  alarm  rate  using  all  parameters  were  29.0  %  and  27.1  % 
respectively,  corresponding  to  18  UXO  and  49  scrap  targets  missed.  The  parameters 
found  to  be  most  important  (causing  the  greatest  increase  in  UXO  miss  rate)  in  the  tests 
using  five-parameter  patterns  were  inc  >  siz  >  fit  >  dep  >  azi  =  mom.  The  parameter  fit  is 
expected  to  improve  the  performance  when  excluded  because  it  is  not  based  on  any 
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Table  5.  MTADS  parameter  combinations  and  overall  PNN  cross  validation  results 
for  BBR  2  Air  Base  data  set  using  four  classes  (2.75”  War  heads  (WH),  M38  bombs, 
SCAR  rockets,  and  Scrap 


Parameter  Combination 

UXO 

FALSE 

#UXO 

#  Scrap 

1 

2 

3 

4 

5 

6 

Miss  Rate  (%) 

Alarm  Rate  (%) 

Missed 

Missed 

dep 

59.7 

14.9 

37 

27 

- 

siz 

- 

- 

- 

- 

41.9 

38.7 

26 

70 

- 

- 

mom 

- 

- 

- 

25.8 

97.2 

16 

176 

_ 

- 

- 

inc 

- 

- 

40.3 

73.5 

25 

133 

- 

_ 

- 

- 

azi 

56.5 

88.4 

35 

160 

- 

- 

- 

- 

- 

fit 

59.7 

75.7 

37 

137 

siz 

inc 

19.4 

30.4 

12 

55 

- 

- 

mom 

inc 

- 

- 

24.2 

30.4 

15 

55 

dep 

- 

- 

inc 

- 

- 

53.2 

34.3 

33 

62 

- 

siz 

mom 

- 

- 

- 

43.5 

38.1 

27 

69 

- 

- 

mom 

- 

- 

fit 

51.6 

22.1 

32 

40 

- 

- 

- 

- 

azi 

fit 

61.3 

69.1 

38 

125 

dep 

siz 

inc 

40.3 

12.2 

25 

22 

- 

siz 

mom 

inc 

- 

- 

22.6 

30.4 

14 

55 

- 

siz 

- 

inc 

azi 

- 

37.1 

13.8 

23 

25 

- 

- 

- 

inc 

azi 

fit 

56.5 

51.9 

35 

94 

dep 

siz 

mom 

- 

- 

- 

30.6 

19.3 

19 

35 

- 

- 

mom 

- 

azi 

fit 

37.1 

38.7 

23 

70 

siz 

inc 

azi 

fit 

30.6 

22.1 

19 

40 

dep 

siz 

mom 

inc 

- 

- 

48.4 

11.6 

30 

21 

- 

siz 

mom 

inc 

azi 

- 

33.9 

16.6 

21 

30 

dep 

inc 

azi 

- 

29.0 

19.3 

18 

35 

- 

- 

azi 

fit 

51.6 

19.9 

32 

36 

siz 

mom 

inc 

azi 

fit 

33.9 

22.7 

21 

41 

dep 

- 

mom 

inc 

azi 

fit 

38.7 

19.9 

24 

36 

dep 

siz 

- 

inc 

azi 

fit 

32.3 

23.2 

20 

42 

dep 

siz 

mom 

- 

azi 

fit 

45.2 

24.3 

28 

44 

dep 

siz 

mom 

inc 

- 

fit 

32.3 

22.1 

20 

40 

dep 

siz 

mom 

inc 

azi 

- 

35.5 

19.3 

22 

35 

dep 

siz 

mom 

inc 

azi 

fit 

29.0 

27.1 

18 

49 

19 


Table  6.  Summary  of  individual  class  miss  rates  (%)  for  representative 
parameter  combinations 


Parameters 

2.75  WH 

Miss  Rate 

(%) 

M38 

SCAR 

2.25 

Scrap 

Overall  UXO 

mom  inc 

57.1 

23.8 

51.9 

40.3 

43.6 

siz  inc 

28.6 

38.1 

44.4 

23.2 

38.7 

dep  siz  inc 

28.6 

9.5 

33.3 

27.6 

24.2 

dep  siz  mom  inc 
azi 

42.9 

4.8 

51.9 

24.9 

33.9 

dep  siz  mom  inc 
azi  fit 

42.9 

4.8 

44.4 

25.4 

24.2 

physical  parameters  of  the  target  being  identified.  However,  in  this  case  it  had  a 
detrimental  effect,  perhaps  due  to  better  modeler  fits  on  UXO  versus  scrap  at  this  site. 

The  best  single  parameter  for  overall  prediction  was  siz  with  a  UXO  miss  rate  of  41.9  % 
and  a  false  alarm  rate  of  28.7.  The  use  of  fit  alone  resulted  in  the  largest  classification 
errors:  59.7  %  and  75.7  %  for  the  UXO  miss  rate  and  false  alarm  rate  respectively.  The 
best  overall  classifications  were  achieved  for  the  three-parameter  sets:  [siz,  inc,  azi]  and 
[siz,  mom,  inc].  Using  only  two  parameters  as  the  input  patterns,  siz  and  inc,  was  the 
combination  that  produced  the  best  UXO  miss  rate,  19.4  %,  and  false  alarm  rate,  30.4  %, 
which  correspond  to  12  UXO  and  55  scrap  targets  missed.  The  best  UXO  detection  was 
obtained  with  the  three-parameter  pattern  containing  siz,  mom,  inc.  The  UXO  miss  rate 
was  22.6  %  and  the  false  alarm  rate  was  30.4  %. 

Shown  in  Figure  11  are  plots  of  the  PNN  CV  predicted  probability  of  being  2.75” 
Warheads,  M38  bombs,  SCAR  rockets,  and  scrap  for  each  of  the  243  patterns  in  the  BBR 
2  data  set  using  three  parameters  (dep  siz  inc).  The  best  results  are  observed  for  the  M3  8. 
The  solid,  horizontal  line  in  each  UXO  plot  represents  an  arbitrary  selection  of  the 
probability  cut  off  threshold  to  be  used  for  this  class  and  data  set.  As  discussed  above, 
the  probability  threshold  can  be  adjusted  in  case  a  priori  information  is  available  or  to 
reduce  the  number  of  missed  detections.  However,  in  several  cases,  missed  detections 
were  predicted  with  very  low  probabilities  of  being  UXO  (<  40  %).  These  cut  off  lines 
were  drawn  with  the  intent  to  prevent  the  missed  detection  of  UXO  at  the  expense  of 
false  alarms.  Fortunately,  the  majority  of  scrap  items  cluster  close  to  zero  in  the  UXO 
probability  plots  thus  minimizing  the  number  of  false  alarms  at  lower  probability  cut  off 
values.  A  summary  of  the  missed  detections  and  false  alarm  rates  is  given  in  Table  7. 

The  misclassifications  and  low  probabilities  occur  because  no  other  UXO  patterns 
in  the  class  have  similar  MAG  model  outputs.  In  the  warhead  class,  the  four  targets  that 
had  low  probabilities  were  target  numbers  2, 4,  9,  and  1 1.  Examining  the  parameters 
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shows  that  target  numbers  2  and  4  had  low  dep  values,  0.04  and  0.08  respectively 
compared  with  an  average  dep  value  for  the  others  of  0.32  ±  0.19.  All  of  the 
misclassifications  were  assigned  to  the  scrap  class  by  the  PNN.  In  the  M38  class,  four 
targets  had  low  probabilities  of  which  three  were  misclassified:  target  numbers  16,  22, 
and  33.  Target  #  22  had  a  high  dep  value  0.62  and  a  low  inc  value,  9,  compared  with  the 
average  of  the  others:  0.46  ±  0.13  and  42.2  ±  23.5.  The  remediation  comment  for  this 
target  was  “M38  deformed”.  Target  #  33  had  a  larger  siz,  0.30  compared  with  0.21  ±  0.03 
for  the  average  of  the  others.  The  remediation  comment  for  this  target  was  “M3 8  + 
SCAR”.  For  the  SCAR  class,  there  were  eight  targets  with  low  probability  that  were 
misclassified:  target  numbers  38, 39,  47, 49,  51,  52,  61,  62.  Targets  number  38  had  a  high 
siz  and  inc,  0.23  and  37  respectively  compared  with  the  average  values  for  the  others, 
0.15  ±  0.06  and  13.2  ±  8.8.  Target  39  and  49  had  low  inc  values,  0  and  2  respectively. 
Targets  51  and  62  had  high  inc  values,  50  and  33  respectively.  In  addition,  targets  52  and 
62  had  low  dep  values,  both  0.06  compared  with  the  average  value  of  the  other  targets, 

0.15  ±0.06. 

In  the  eventual  application  of  this  technology,  larger  data  sets  comprised  of  UXO 
and  scrap  from  many  sites  will  be  used  to  prevent  this  situation  from  occurring.  An 
extensive  examination  of  the  effects  of  using  different  combinations  of  modeler  output 
parameters  on  the  classification  performance  of  a  PNN  has  been  completed.  The  results 
obtained  in  this  study  are  probably  data  set  dependent  and  may  vary  according  to  the  type 
of  UXO  and  scrap  encountered,  the  amount  of  scrap  present,  soil  type,  and  other 
geological  effects. 

Table  7.  Summary  of  missed  detection  and  false  alarm  rates  for  the  4-class  problem 


Class 

Probability  #  UXO 

Cut  off  (%)  Misclassified 

UXO  Miss 
Rate  (%) 

#  False 
Alarms 

False  Alarms  (%)a 

2.75"  Warheads  (WH) 

4 

1 

7.1 

62 

37.1 

M38  Bombs 

8 

0 

0.0 

10 

6.3 

SCAR 

2 

2 

7.4 

74 

48.1 

Percentage  of  scrap  and  UXO  other  than  the  type  in  question 


Improved  Classification  with  Fit  Parameter  Selection 

Improvements  in  PNN  classification  can  be  obtained  through  the  use  of  data 
selection.  The  selection  of  targets  based  upon  the  quality  of  the  modeler’s  fit  can  provide 
some  discrimination  between  low  and  high  quality  data.  This  approach  can  be  used  to 
construct  a  better  training  set  or  to  select  those  targets  whose  fit  parameter  is  low  enough 
that  positive  identification  is  unlikely.  The  effects  of  retaining  only  those  targets  with  a  fit 
parameter  above  a  specified  cut  off  threshold  were  examined  for  the  two  and  four  class 
problems. 
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BBR  2  Classification  (2  Class  Problem) 

The  results  of  PNN  classification  with  varying  numbers  of  targets  removed  based 
upon  the  fit  parameter  are  given  in  Figure  12.  The  results  of  PNN-CV  on  parameters  1 :5 
(dep  siz  mom  inc  azi)  shown  in  Figure  12  indicates  that  the  best  overall  classification  can 
be  obtained  using  a  fit  parameter  cut  off  of  0.96.  Using  only  targets  with  a  fit  parameter 
>  0.96  results  in  cross-validation  miss  rates  (%)  of  25.6  and  7.5  for  UXO  and  scrap 
respectively.  Figure  13  a  and  b  show  that  at  the  selected  cut  off  (0.96),  47  UXO  targets 
out  of  62  total  UXO  targets  (76%)  were  kept  in  the  data  set.  Similarly,  93  scrap  targets 
out  of  181  total  scrap  targets  (51%)  were  kept  in  the  data  set.  This  represents  a 
compromise  between  the  number  of  targets  removed  and  the  classification  performance. 
It  can  be  seen  in  Figure  12,  that  the  performance  of  both  the  UXO  and  scrap  classes 
improved  somewhat  inconsistently  until  the  fit  parameter  cut  off  was  ~  0.96. 

The  combination  of  the  best  parameters  and  the  fit  parameter  cut  off  did  not  result 
in  further  enhancement  of  performance.  Using  the  combination  [dep  siz  inc]  and  a  fit 
parameter  cutoff  of  0.96  resulted  in  a  UXO  miss  rate  of  17.0%  and  a  scrap  miss  rate  of 
8.6  %.  For  UXO  detection  this  result  is  poorer  than  the  best  parameter  selection,  but 
better  than  the  fit  cut  off  selection.  While  for  scrap  detection  this  result  is  an 
improvement  over  the  best  parameter  selection  result  and  marginally  worse  than  the  fit 
cut  off  selection  result. 


Figure  12.  PNN  cross-validation  miss  rate  (%)  versus  fit  parameter 
cut  off  for  the  2-class  problem 
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Figure  13.  PNN  cross-validation  results  as  a  function  of  fit  parameter  cut  off.  a)  #  UXO 
retained,  b)  #  Scrap  retained 


BBR  2  Classification  (4  Class  Problem) 

The  results  of  PNN  classification  with  varying  numbers  of  targets  removed  based 
upon  the  fit  parameter  are  given  in  Figure  14.  The  results  of  PNN-CV  on  parameters  1 :5 
(dep  siz  mom  inc  azi)  shown  in  Figure  14  indicate  that  the  best  overall  classification  can 
be  obtained  using  a  fit  parameter  cut  off  of  0.92.  Using  only  targets  with  a  fit  parameter 
>  0.92  results  in  cross-validation  miss  rates  (%)  of  25.0,  4.8,  48.1,  and  20.4  for  2.75 
warheads,  M38  bombs,  SCAR  rockets,  and  scrap  respectively.  Figure  13  a  and  b  shows 
that  at  the  selected  cut  off  (0.92),  60  UXO  targets  out  of  62  total  UXO  targets  (97  %) 
were  kept  in  the  data  set.  Similarly,  142  scrap  targets  out  of  181  total  scrap  targets  (79  %) 
were  kept  in  the  data  set.  It  can  be  seen  in  Figure  14,  that  the  classification  performance 
of  all  UXO  classes  (2.75  warheads,  M38  bombs,  and  SCAR  rockets)  improved  or 
remained  constant  with  increasing  fit  parameter  cut  off  (up  to  0.92),  while  class  4  (Scrap) 
classification  worsened  slightly. 

The  combination  of  the  best  parameters  and  the  fit  parameter  cut  off  did  not  result 
in  further  enhancement  of  performance.  Using  the  combination  [dep  siz  inc]  resulted  in  a 
UXO  miss  rate  of  25.0%,  19.0%,  37.0%  for  2.75  warheads,  M38  bombs,  and  SCAR 
rockets,  respectively,  and  a  scrap  miss  rate  of  21.1%.  For  2.75  warheads,  the 
classification  result  was  the  same,  while  the  classification  was  poorer  for  M38  bombs. 
There  was  some  improvement  in  the  classification  of  SCAR  rockets,  but  a  slight  increase 
in  the  scrap  miss  rate. 
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Figure  14.  PNN  cross-validation  miss  rate  (%)  versus  fit  parameter 
cut  off  for  the  four-class  problem 


Former  Buckley  Air  Base 

The  Buckley  data  set  (Buckley  Air  Base,  Bombing  Target  #2,  Section  A5)  was 
investigated  to  determine  the  ability  of  a  PNN  to  distinguish  UXO  from  scrap  at  this  site. 
For  PNN  classification,  the  Buckley  data  set  tested  contained  MAG  physics-based 
modeler  outputs  from  392  buried  objects  that  had  been  remediated.  A  392  x  6  data  matrix 
was  generated  and  evaluated  as  both  a  two-  and  five-class  problem.  The  pattern  vectors 
were  subdivided  into  two  categories  or  classes,  UXO  and  Scrap  for  the  data  set 
classification  labeled  Buckleyl  and  into  five  classes  for  the  data  set  classification  labeled 
Buckley2.  The  Buckley2  data  set  consisted  of  69  M38,  8  SCAR  rockets,  7  M69  bombs, 
30  MK23  practice  bombs  and  6  M100  series  fuses  (in  the  same  class),  while  the  272 
scrap  primarily  consisted  of  OE  scrap  and  scrap  metal.  A  few  (target  #  292,  339,  340)  of 
the  targets  did  not  have  modeler  outputs  in  the  table  and  three  others  had  descriptions 
which  were  unclear  and  were  therefore  left  out  of  the  data  set. 

The  data  sets  used  for  classification  testing  were  prepared  by  selecting  those 
targets  with  the  descriptions  listed  above  from  the  modeler  output  and  truth  tables  for  this 
site.  The  PNN  was  used  to  discriminate  either  two  classes  or  four  classes  of  objects  (i.e., 
differentiate  four  UXO  types  from  each  other  and  scrap).  The  parameters  used  in  this 
work  were  derived  from  MAG  data.  The  Buckley  data  set  represents  a  challenge  to  the 
PNN  classification  routines  using  modeler  outputs  for  the  discrimination  of  UXO  from 
scrap. 
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Data  Set  Characterization 


An  examination  of  the  data  set  using  PC  A  and  non-linear  mapping  was 
undertaken  to  understand  the  clustering  and  overlap  of  the  UXO  classes  and  scrap  in  the 
data  set.  The  scores  from  the  first  two  factors,  or  principal  components,  for  the  Buckley 
data  set  are  shown  Figure  15.  Significant  overlap  between  the  classes  can  be  seen  in  this 
plot.  However,  there  is  some  separation  between  the  classes  indicating  differences  in  the 
modeler  outputs  for  each  class.  A  plot  of  the  data  in  two  dimensions  generated  by  non¬ 
linear  mapping  is  shown  in  Figure  16.  An  examination  of  Figure  16  reveals  that  the  scrap 
class  is  overlapped  with  the  MK23  class.  Based  upon  the  degree  of  separation  between 
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Figure  15.  PCA  scores  plot  of  the  Buckley  data  set 

the  classes  seen  in  both  PCA  and  non-linear  mapping,  accurate  classification  of  this  data 
set  should  be  challenging.  One  target  (target  #  84)  is  an  outlier  in  PCA  and  non-linear 
mapping  found  at  13.5,  -2.9  and  14.0,  -9.0  respectively.  The  description  in  the 
remediation  list  was  incendiary  cluster  bomb  and  the  source  of  the  error  was  likely  the 
moment  parameter  which  had  a  value  of  32.3.  This  moment  is  very  high  for  such  an  item: 
the  average  of  others  in  its  class  was  1.5  ±  0.5. 
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Figure  16.  Non-linear  map  plot  of  the  Buckley  data  set 


Cross-Validation  Performance 

PNN  classification  performance  was  examined  using  leave-one-out-cross¬ 
validation  (CV).  In  this  work,  the  PNN  returned  either  four  or  two  values  for  each 
pattern:  the  probability  of  being  one  of  the  three  UXO  types  and  the  probability  of  being 
scrap  or  simply  UXO  or  scrap.  Each  pattern  was  assigned  to  the  category  having  the 
highest  probability.  In  a  real-world  scenario,  this  probability  threshold  can  be  adjusted  for 
a  particular  site  using  a  priori  information  regarding  the  types  and  relative  number  of 
buried  objects. 

To  determine  which  MAG  model  parameters  were  most  important  and  to  gain 
insight  into  the  PNN  classification,  the  effect  of  various  model  parameters  were  studied. 
PNN  CV  performance  was  determined  for  all  possible  combinations  of  one,  two,  three, 
four,  five,  and  six  MAG  model  parameters.  The  UXO  miss  rate  (missed  detection)  is 
defined  as  the  ratio  of  the  number  of  UXO  classified  as  scrap  to  the  total  number  of  UXO 
targets  classified.  The  false  alarm  rate  is  defined  as  the  ratio  of  the  number  of  scrap 
classified  as  UXO  to  the  total  number  of  scrap  classified. 
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Buckley2  Classification  (Five-Class  Problem) 

The  overall  results  from  the  single,  five  and  six  parameter  patterns  and 
representative  results  from  the  two,  three,  and  four  parameter  patterns  are  given  in  Table 
8.  The  UXO  miss  rate  and  false  alarm  rate  using  all  parameters  were  34.2  %  and  17.3  % 
respectively,  corresponding  to  41  UXO  and  47  scrap  targets  missed  for  an  overall  correct 
classification  of  78  %  of  the  data  set.  The  parameters  found  to  be  most  important 
(causing  the  greatest  increase  in  UXO  miss  rate)  in  the  tests  using  five-parameter  patterns 
were  dep  =  inc  >  siz  >  fit  >  azi.  One  of  the  parameters,  azi  had  a  beneficial  effect  on  the 
UXO  miss  rate  when  left  out  of  the  pattern.  The  omission  of  the  parameter,  mom,  had  a 
beneficial  effect  on  the  classification  of  this  data  set  which  is  different  from  its  effect 
when  omitted  from  the  Badlands  data  set  (no  effect). 

There  was  no  single  best  parameter  for  overall  prediction:  all  had  poor 
classification  or  fell  prey  to  over  training  as  indicated  by  the  superscript  “a”  in  Table  8. 
The  best  results  for  UXO  detection  were  obtained  with  three  paramters  [dep  siz  inc],  and 
five  parameters  [dep  siz  mom  inc  azi]  and  [dep  siz  inc  azi  fit].  The  UXO  miss  rate  for 
three  parameter  combination  was  29.2  %  and  the  false  alarm  rate  was  24.3  %.  The 
individual  miss  rate  percentages  for  each  class  are  given  in  Table  9  for  the  best  paiameter 
combination  and  compared  with  the  result  obtained  using  all  parameters. 
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Table  8.  MTADS  parameter  combinations  and  overall  PNN  cross  validation  results 
for  Buckley  Air  Base  data  set  (5  class  problem) 


Parameter  Combination 

UXO 

False 

#UXO 

#  Scrap 

1 

2 

3 

4 

5 

6 

Miss  Rate  (%) 

Alarm  Rate  (%) 

Missed 

Missed 

dep 

. 

. 

62.5 

33.1 

75 

90 

- 

siz 

- 

- 

- 

- 

23.3 

34.9 

28 

95 

_ 

_ 

mom 

- 

- 

- 

0.0 

4.4 

0 

12 

- 

- 

- 

inc 

- 

- 

75.0 

63.6 

90 

173 

_ 

_ 

- 

- 

azi 

65.0 

76.8 

78 

209 

- 

_ 

- 

- 

- 

fit 

83.3 

40.4 

100 

110 

siz 

inc 

40.8 

27.2 

49 

74 

- 

- 

mom 

inc 

- 

- 

48.3 

27.2 

58 

74 

dep 

- 

- 

inc 

- 

- 

43.3 

37.5 

52 

102 

- 

siz 

mom 

- 

- 

- 

45.8 

27.6 

55 

75 

- 

- 

mom 

- 

- 

fit 

46.7 

33.1 

56 

90 

« 

- 

_ 

- 

azi 

fit 

68.3 

50.7 

82 

138 

dep 

siz 

inc 

29.2 

24.3 

35 

66 

- 

siz 

mom 

inc 

- 

- 

41.7 

27.2 

50 

74 

- 

siz 

- 

inc 

azi 

- 

39.2 

28.3 

47 

77 

- 

- 

- 

inc 

azi 

fit 

54.2 

47.4 

65 

129 

dep 

siz 

mom 

- 

- 

- 

43.3 

30.1 

52 

82 

_ 

- 

mom 

- 

azi 

fit 

50.8 

30.1 

61 

82 

siz 

inc 

azi 

fit 

41.7 

24.3 

50 

66 

dep 

siz 

mom 

inc 

- 

- 

31.7 

24.6 

38 

67 

- 

siz 

mom 

inc 

azi 

- 

41.7 

27.9 

50 

76 

dep 

siz 

- 

- 

azi 

fit 

40.0 

22.1 

48 

60 

- 

siz 

mom 

- 

azi 

fit 

46.7 

29.8 

56 

81 

siz 

mom 

inc 

azi 

fit 

40.0 

21.7 

48 

59 

dep 

- 

mom 

inc 

azi 

fit 

35.8 

21.7 

43 

59 

dep 

siz 

- 

inc 

azi 

fit 

30.8 

19.1 

37 

52 

dep 

siz 

mom 

- 

azi 

fit 

40.0 

21.3 

48 

58 

dep 

siz 

mom 

inc 

- 

fit 

32.5 

22.1 

39 

60 

dep 

siz 

mom 

inc 

azi 

- 

35.0 

25.0 

42 

68 

dep 

siz 

mom 

inc 

azi 

fit 

34.2 

17.3 

41 

47 

a  Good  result  is  due  to  over  training  not  intrinsic  properties  of  the  data  and  represents  an  anomaly. 
Over  training  indicated  by  large  sigmas:  1 .35  and  8.40  respectively 
compared  with  average  of  others:  0.34  ±  0.10 
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Table  9.  Best  Class  Miss  Rate  Percentages 


Parameters 

Miss  Rate  (%)*  Overall  UXO 

Class  1  Class  2  Class  3  Class  4  Class  5  Miss  Rate  (%) 

siz 

inc 

40.6 

50.0 

57.1 

36.1 

26.8 

40.8 

mom 

inc 

42.0 

50.0 

100.0 

50.0 

27.2 

45.8 

siz 

mom 

47.8 

75.0 

57.1 

33.3 

27.6 

48.3 

dep 

siz  inc 

23.2 

25.0 

42.9 

36.1 

24.3 

29.2 

dep 

siz  mom  inc 

24.6 

37.5 

42.9 

36.1 

24.3 

31.7 

All  parameters 

26.1 

25.0 

71.4 

44.4 

17.3 

34.2 

*  Class  1  are  M38,  class  2  are  SCAR,  class  3  are  M69  bombs,  class  4  are  MK23  practice  bombs  and 
M100  series  fuses,  and  class  5  are  scrap. 


MK23& 

MlOO 

M38  SCAR  M69  fuse  SCRAP 


M38 


SCAR 

M69 

MK23& 

MlOO 

fuse 

SCRAP 


Classification 
Correct  (%) 

76.8 

75.0 

57.1 

63.9 

75.7 


Figure  17.  Confusion  matrix  showing  the  individual  misclassifications  in  each  class  for  the  5 
class  problem  using  the  3  parameters  [dep  siz  inc]  found  to  be  one  of  the  most  important 
combinations  for  good  classification 
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The  PNN-CV  classification  results  are  shown  in  Figure  17  for  each  class  using  one 
of  the  better  parameter  combinations:  [dep  siz  inc].  The  results  are  displayed  in  a 
confusion  matrix  format  to  illustrate  the  misclassifications.  It  should  be  noted  that 
classification  percentages  listed  in  the  figure  are  for  the  individual  classes.  If  one 
considers  UXO  targets  misclassified  as  other  UXO,  the  percentage  increases  for  M3  8, 
M69,  and  MK23  &  mlOO  fuses:  95.6  %,  85.7  %,  75.0  %  respectively. 

Shown  in  Figure  18  is  a  plot  of  the  PNN  CV  predicted  probability  of  being  one  of  the 
four  UXO  classes  for  each  of  the  392  patterns  in  the  Buckley  data  set  using  just  three 
MAG  parameters  (dep,  siz,  inc).  The  upper  solid,  horizontal  line  represents  an  arbitrary 
selection  of  the  probability  cut  off  threshold  (20  %)  to  be  used  for  this  data  set.  As 
discussed  above,  the  probability  threshold  can  be  adjusted  in  case  a  priori  information  is 
available  or  to  reduce  the  number  of  missed  detections.  However,  in  several  cases, 
missed  detections  were  predicted  with  very  low  probabilities  of  being  UXO  (<  5  %). 
These  misclassifications  occur  because  no  other  UXO  patterns  in  the  class  have  similar 
MAG  model  outputs.  Using  a  20  %  cut  off,  28  UXO  were  missed  (23  %)  with  a  false 
alarm  rate  of  9  %.  Further  improvements  in  UXO  detection  can  be  obtained  by  lowering 
the  probability  threshold  further.  At  10  %  cut  off,  21  UXO  were  missed  (18  %)  with  a 
false  alarm  rate  of  13  %.  Finally,  at  5  %  cut  off,  15  UXO  were  missed  (13  %)  with  a  false 
alarm  rate  of  17  %. 


Target  # 


Figure  18.  Probability  of  being  a  UXO  (four  UXO  classes)  versus  target  number. 
The  first  120  targets  are  UXO. 
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Buckley  1  Classification  (2  Class  Problem) 

Representative  results  from  the  two,  three,  four  and  five  parameter  patterns  are 
given  in  Table  10.  The  UXO  miss  rate  and  false  alarm  rate  using  all  parameters  were 
12.5%  and  14.0%  respectively,  corresponding  to  15  UXO  and  38  scrap  targets  missed. 
Using  fewer  parameters  did  not  benefit  classification  of  UXO.  Some  benefit  was  seen  for 
scrap  classification  using  the  two  parameter  combinations.  Using  only  two  parameters  as 
the  input  patterns,  [siz  inc]  produced  19.2%  UXO  miss  rate,  and  12.1%  false  alarm  rate. 
The  increase  in  the  UXO  miss  rate  with  reduced  parameters  is  markedly  different  from 
the  other  data  sets  studied  where  the  classification  performance  of  the  two  class  problem 
remained  the  same  or  improved. 

Table  10.  MTADS  parameter  combinations  and  PNN  cross  validation  results 
for  Buckley  Air  Base  data  set  (2  Class  Problem) 


Parameters 

Misclassification 

rate 

UXO  (%) 

Scrap  (%) 

siz 

inc 

19.2 

12.1 

mom 

inc 

34.2 

7.4 

siz 

mom 

30.0 

12.9 

dep 

siz 

inc 

19.2 

15.1 

dep 

siz 

mom 

inc 

19.2 

15.1 

dep 

siz 

mom 

inc 

azi 

18.3 

15.1 

All  parameters 

12.5 

14.0 

The  probability  of  being  a  UXO  is  shown  in  Figure  19  for  all  392  targets  in  the 
Buckley  data  set.  These  probabilities  were  generated  using  the  parameter  combination 
found  to  produce  the  best  classification  results:  [dep  siz  inc].  The  solid,  horizontal  line 
represents  an  arbitrary  selection  of  the  probability  cut  off  threshold  (30  %)  to  be  used  for 
this  class  and  data  set.  As  discussed  above,  the  probability  threshold  can  be  adjusted  in 
case  a  priori  information  is  available  or  to  reduce  the  number  of  missed  UXO.  Below  the 
30  %  probability  cut  off,  marked  by  the  upper  solid  horizontal  line,  11  UXO  targets  (9.2 
%)  were  missed  and  70  scrap  items  (25.7  %)  were  identified  as  UXO  (false  alarm).  The 
identities  of  the  misclassified  UXO  targets  are  given  in  Table  1 1  with  the  possible  cause 
of  misclassification.  The  outlier  (target  #  84,  with  mom  =  32.3)  was  correctly  classified  in 
this  case  because  mom  was  not  used.  To  reduce  the  number  of  missed  UXO,  the 
probability  cut  off  can  be  reduced  to  20  %,  allowing  detection  of  an  additional  4  UXO. 
The  cost  of  the  additional  4  UXO  detections  is  an  increase  in  the  number  of  false  alarms: 
92  false  alarms  versus  70  false  alarms  at  30  %  probability  cut  off.  These  amount  to  a 
UXO  miss  rate  of  5.8  %  and  a  false  alarm  rate  of  33.8  %.  The  ROC  curve  for  the  Buckley 
data  set  can  be  seen  in  Figure  20. 
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Table  11.  Buckley  2  class  problem  using  [dep  siz  inc]  misclassified 
targets  and  possible  cause 


Target  # 

Description 

Possible 

Misclassification  Cause 

52 

M38 

low  dep,  siz 

54 

M38 

low  siz 

82 

AN-M69  bomblet  cluster 

low  inc 

90 

MK23 

low  inc 

104 

MK23  (borderline) 

low  siz 

107 

MK23 

low  dep 

108 

MK23 

low  dep 

112 

MK23 

low  siz 

114 

MK23  Practice  bomb 

low  dep 

115 

100  Series  fuse 

low  dep,  siz 

Target  # 

Figure  19.  Probability  of  being  UXO  versus  target  number  for  the  two  class 
problem  in  the  Buckleyl  data  set  using  [dep  siz  inc].  The  first  120  targets  are 
UXO 
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Figure  20.  Receiver  operator  characteristic  curve  for  Buckley 


An  extensive  examination  of  the  effects  of  using  different  combinations  of 
modeler  output  parameters  on  the  classification  performance  of  a  PNN  has  been 
completed.  The  results  obtained  in  this  study  are  probably  data  set  dependent  and  may 
vary  according  to  the  type  of  UXO  and  scrap  encountered,  the  amount  of  scrap  present, 
soil  type,  etc.  This  is  verified  by  comparing  the  results  obtained  using  the  different  data 
sets.  The  improvement  in  miss  rate  was  3.4  %  for  the  Buckley2  data  set.  The  overall 
improvement  in  the  UXO  miss  rate  for  the  BBR  1  data  set  was  only  2  %  and  there  was  no 
false  alarm  penalty  for  the  improved  UXO  detection  capability.  For  the  two  class  problem 
in  BBR  2,  there  was  a  greater  improvement  (11  %)  in  UXO  miss  rate,  with  only  a  small 
false  alarm  penalty  (0.6  %).  In  the  Buckley  data  set,  classification  was  adversely  effected 
by  parameter  reduction  and  selection  in  the  2  class  problem  and  the  best  overall 
performance  obtainable  was  with  all  parameters  using  two  classes.  However,  reasonable 
classification  could  be  achieved  using  a  similar  parameter  selection  as  that  used  for  the 
other  data  sets  [dep  siz  inc]  or  [siz  inc]. 
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Analysis  of  M38  Targets  from  BBR1  and  BBR  2 

Principal  component  analysis  was  performed  on  a  data  set  prepared  from  M3  8 
targets  measured  and  remediated  at  BBR  1  and  BBR  2  to  determine  if  the  targets 
observed  at  different  locations  at  this  site  clustered  similarly.  All  of  the  scrap  items  from 
both  sites  were  included  in  the  analysis.  One  can  see  in  the  PC  A  plot  shown  in  Figure  21 
that  the  BBR1  and  BBR  2  M38  targets  are  overlapped  and  separated  from  the  scrap  found 
at  both  locations.  This  indicates  that  either  data  set  could  be  used  as  a  training 


PC  1 

Figure  21.  Principal  component  analysis  of  M38  targets  from  BBR  1  and 
BBR  2.  Two  principal  components  account  for  61.9  %  of  the  variance 

set  to  predict  the  M38  targets  in  the  other.  The  ability  to  use  data  from  one  location  to 
predict  items  at  another  location  is  of  utmost  importance  to  the  development  of  the  PNN. 
Ideally,  training  data  from  one  location  could  be  used  at  least  as  a  starting  point  for  an 
entirely  different  site.  In  an  effort  to  test  this,  the  entire  BBR  2  data  set  was  used  to  train 
the  PNN  and  the  BBR1  data  set  was  used  in  prediction.  The  parameters  used  were  those 
that  gave  the  best  results  in  PNN-CV  for  BBR  2:  [dep  siz  inc].  For  the  two  classes  in 
BBR1,  M38  and  scrap,  the  UXO  miss  rate  was  4.5  %  and  the  false  alarm  rate  was  34.9 
%,  using  a  40  %  probability  cutoff.  A  plot  of  the  probability  of  being  an  M38  versus 
target  number  is  shown  in  Figure  22.  There  are  only  two  UXO  targets  with  probabilities 
below  40  %.  These  targets,  19  and  34,  were  also  misclassified  during  the  PNN-CV  of  the 
BBR1  data  set.  The  ROC  curve  for  this  prediction  set  is  shown  in  Figure  23. 
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Figure  22.  Probability  of  being  M38  versus  target  number  for  BBR  2 
training  set  with  BBR  1  prediction  set 


False  Alarm  rate 


Figure  23.  Receiver  operator  curve  for  BBR  2  training  set  with  BBR  1 
prediction  set 


CONCLUSIONS 


This  report  demonstrates  the  usefulness  of  multivariate  methods  for  identifying 
UXO  in  the  presence  of  scrap  and  other  clutter.  In  particular,  the  methods  have  been  used 
to  select  the  optimal  set  of  MAG  modeler  parameters  to  achieve  the  best  classification 
results  for  three  different  UXO  surveys.  Using  the  Badlands  Bull’s  eye  Target  1,  94  %  of 
the  targets  are  correctly  identified,  and  86  %  correct  classification  was  achieved  at  the 
more  complex  Target  2  site.  The  Buckley  site  was  the  most  difficult  using  only  the  MAG 
modeler  outputs  due  to  the  small  size  of  the  UXO  targets  and  only  84  %  of  the  targets  are 
correctly  identified.  The  ability  to  use  the  classification  model  from  one  site  on  data  from 
another  was  demonstrated  for  the  two  Badlands  Test  Sites.  The  PNN  classifier  was 
trained  using  the  Badlands  Target  2  dataset,  and  was  able  to  correcdy  predict  93  %  of  the 
targets  at  Bull’s  eye  Target  1. 

Three  MTADS  data  sets  have  been  evaluated  and  classification  performance  tested 
using  a  PNN.  Several  approaches  to  improve  the  performance  of  the  classification  have 
been  investigated.  These  include  variable  or  parameter  selection  based  upon  cross- 
validation  testing,  probability  threshold  reduction,  and  data  exclusion  via  fit  parameter 
cut  off.  Of  these  approaches,  parameter  selection  and  probability  threshold  reduction 
were  found  to  be  the  most  useful.  The  variables  found  to  be  most  important  were  siz,  inc, 
dep,  and  mom  in  order  of  importance.  For  BBR1,  [siz,  inc]  were  the  favored 
combination;  for  BBR2,  [siz,  inc,  azi]  were  the  variables  that  produced  the  best  results, 
with  [dep,  siz,  inc]  being  nearly  as  effective.  For  the  Buckley  data  set,  the  most  important 
variables  were  [dep  siz  inc].  For  the  BBR2  and  Buckley  data  sets,  two  classification  types 
were  attempted:  2  class  (UXO  and  scrap)  and  multi-class  (UXO  types  and  scrap).  In  both 
cases,  better  classification  (less  UXO  missed)  was  obtained  by  defining  the  problem  in 
terms  of  two  classes. 

For  BBR  1,  the  improvement  using  variable  selection  was  2.3  %,  while  no 
improvement  was  found  using  probability  threshold  reduction.  At  BBR  2,  there  was 
additional  11.3  %  improvement  in  classification  using  variable  selection  and  an  8.1  %  to 
11.3  %  improvement  over  that  using  probability  threshold  reduction.  There  was  no 
improvement  at  the  Buckley  site  using  variable  selection  and  a  13.4  %  improvement 
using  probability  threshold  reduction  (for  two  class  problem  with  [dep  siz  inc]). 

It  should  be  noted  that  the  fit  parameter  cut  off  would  only  be  useful  for  training 
set  selection.  If  the  fit  parameter  was  too  low,  then  the  target  would  automatically  be 
remediated  otherwise  the  PNN  would  classify  the  target.  The  performance  benefits  seen 
using  the  fit  parameter  cut  off  were  inconsistent  and  no  clear  trend  in  the  performance  as 
a  function  of  fit  parameter  was  detectable.  Furthermore,  the  combination  of  the  fit 
parameter  cut  off  and  variable  selection  did  not  generally  have  a  beneficial  synergistic 
effect. 

A  summary  of  the  classification  results  for  all  three  datasets  investigated  is  given 
in  Table  12.  As  the  probability  threshold  is  decreased,  the  number  of  UXO  missed 
detections  decreases  and  the  scrap  false  alarms  increases.  At  BBR  1,  placing  the 
probability  threshold  below  50  %  does  not  permit  the  correct  classification  of  any 
additional  UXO  or  scrap  targets  without  an  unacceptable  increase  in  false  alarms. 
Improvements  were  possible  at  BBR  2  by  lowering  the  threshold  from  50  %  to  10  % 
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where  none  of  the  UXO  were  misclassified.  At  Buckley,  a  probability  threshold  of  20  % 
yielded  7  UXO  misclassified.  Even  with  a  probability  threshold  of  10  %  three  UXO 
targets  were  still  misclassified  while  requiring  almost  all  of  the  scrap  to  be  remediated. 


Table  12.  Summary  of  the  best  UXO  and  scrap  classification  rates  at  various 
probability  thresholds 


Modeler 

Output 

Probability 

UXO 

Scrap 

Overall 

Dataset 

Parameters 

Coding3 

Threshold  (%) 

Correct 

Correct 

Classification  (%) 

Badlands  Target  1 

[dep  siz  inc] 

2 

50 

42/44 

40/43 

94.3 

b 

b 

b 

b 

Badlands  Target  2 

[dep  siz  inc] 

2 

50 

49/62 

160/181 

86.0 

20 

61/62 

112/181 

71.2 

10 

62/62 

81/181 

58.9 

Buckley 

[dep  siz  inc] 

2 

50 

97/120 

231/272 

83.7 

30 

109/120 

202/272 

79.3 

20 

113/120 

180/272 

74.7 

3  Number  of  classes  used  in  training  of  PNN 

b  No  additional  benefits  obtained  through  reduction  of  probability  threshold 


The  performance  of  any  classification  routine,  including  the  PNN,  is  limited  by 
the  separation  of  the  data  in  multi-dimensional  space.  The  PNN  is  known  for  its  ability  to 
accommodate  and  correctly  classify  multi-modal  data  sets.  It  is  our  recommendation  that 
improvements  to  the  modeler  or  inclusion  of  EM  data  will  be  needed  for  better 
classification  of  highly  overlapping  data  sets. 

Future  work  will  also  include  a  detailed  investigation  of  how  well  the  PNN  will 
perform  when  trained  using  data  from  one  site  to  predict  another.  The  initial  results  using 
BBR  1  and  BBR  2  presented  in  this  report  indicate  that  this  will  be  possible.  Other  work 
to  improve  the  classification  performance  will  include  the  use  of  multiple  kernel  widths 
(a)  in  the  PNN.  Incorporation  of  the  EM  data  into  these  models  is  expected  to  improve 
the  classification  results. 
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